Skip to content

Add CI workflow for updating regression test baselines#321

Merged
harryswift01 merged 2 commits intomainfrom
320-feature-automate-regression-baseline
Apr 29, 2026
Merged

Add CI workflow for updating regression test baselines#321
harryswift01 merged 2 commits intomainfrom
320-feature-automate-regression-baseline

Conversation

@harryswift01
Copy link
Copy Markdown
Member

@harryswift01 harryswift01 commented Apr 27, 2026

Summary

This PR will add a GitHub Actions workflow to automate updating regression testing baselines in a consistent CI environment.

Changes

Add regression baseline update workflow:

  • Create a new workflow triggered via workflow_dispatch
  • Run regression tests with --update-baselines flag in CI
  • Configure workflow to run on a stable Ubuntu environment with a fixed Python version

Enable automated commits from CI:

  • Grant contents: write permission to allow committing changes
  • Add auto-commit step using git-auto-commit-action
  • Use Conventional Commit message: test(regression): update baselines
  • Configure commits to use github-actions[bot] identity

Improve baseline consistency:

  • Ensure baselines are generated in a consistent environment
  • Remove dependency on developer local environments
  • Reduce noisy diffs caused by path differences

Impact

  • Reduces noise in PRs caused by local baseline updates
  • Makes regression baseline updates reproducible and consistent
  • Simplifies contributor workflow for updating baselines
  • Keeps test artifacts aligned with CI environment

@harryswift01 harryswift01 added this to the 2.1.2 milestone Apr 27, 2026
@harryswift01 harryswift01 requested a review from jimboid April 27, 2026 15:46
@harryswift01 harryswift01 self-assigned this Apr 27, 2026
@harryswift01 harryswift01 added feature request New feature or request github_actions Pull requests that update GitHub Actions code labels Apr 27, 2026
@harryswift01 harryswift01 linked an issue Apr 27, 2026 that may be closed by this pull request
[Feature]: Automate regression baseline updates via GitHub Actions #320
Closed
@skfegan
Copy link
Copy Markdown
Member

skfegan commented Apr 27, 2026

Improving the baseline changes is great. Do we need to update the documentation to explain how to use this new workflow to update the baselines? And explain clearly when updating the baselines is accepted rather than people just doing this every time a regression test fails?

@harryswift01
Copy link
Copy Markdown
Member Author

harryswift01 commented Apr 27, 2026

Improving the baseline changes is great. Do we need to update the documentation to explain how to use this new workflow to update the baselines? And explain clearly when updating the baselines is accepted rather than people just doing this every time a regression test fails?

Yeah that's a great idea, I'll update the docs to reflect these changes as well.

jimboid
jimboid approved these changes Apr 29, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@jimboid jimboid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR looks great and should cut down on the noisey PRs when baseline updates are involved.

@harryswift01 harryswift01 merged commit fb3bba6 into main Apr 29, 2026
44 of 45 checks passed
@harryswift01 harryswift01 deleted the 320-feature-automate-regression-baseline branch April 29, 2026 12:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jimboid jimboid jimboid approved these changes

Assignees

@harryswift01 harryswift01

Labels

feature request New feature or request github_actions Pull requests that update GitHub Actions code

Projects

None yet

Milestone

2.1.2

Development

Successfully merging this pull request may close these issues.

[Feature]: Automate regression baseline updates via GitHub Actions

3 participants

@harryswift01 @skfegan @jimboid